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REFERENCE TO RELATED APPLICATION 

This Application is a Continuation-In-Part of U.S. Patent Application Serial No. 
10 08/754,729, entitled Integrated Virtual Networks, filed 03/26/97, by W. Benman. 



BACKGROUND OF THE INVENTION 



Field of the Invention: 



The present invention relates to image processing systems and techniques. More 
specifically, the present invention relates to systems and techniques for transplanting one 
20 image of an object from one scene into another scene. 

While the present invention is described herein with reference to illustrative 
embodiments for particular applications, it should be understood that the invention is not 
limited thereto. Those having ordinary skill in the art and access to the teachings 
provided herein will recognize additional modifications, applications, and embodiments 
25 within the scope thereof and additional fields in which the present invention would be of 
significant utility. 
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Description of the Related Art : 

For many applications, there is often a need to extract an image of an object from 
a background scene and transplant that image into another scene. In television and film 
5 production, this is typically accomplished by capturing the image of the desired object in 
a monochromatic background such as a 'blue screen'. Then, by simply color filtering the 
captured image the image of the desired object may be isolated and transplanted into or 
superimposed onto a desired background. 

While this approach is satisfactory for many studio applications, there is a need to 
10 achieve image transplantation without requiring use of the monochromatic background. 
For example, U. S. Patent No. 5784546, entitled Integrated Virtual Networks issued July 
21, 1998 to W. J. Benman, the teachings of which are incorporated herein by reference, 
discloses and claims a computer-based system which allows a user to see a realistic three- 
dimensional representation of an environment, such as an office, on a computer screen. 
15 Real world functionality is mapped onto numerous objects in the environment allowing 
the user to use the objects in the environment (e.g., computer, desk, file cabinets, 
documents, etc.) in same manner as the objects would be used in the real world. 

In addition, Benman' s system allows the user to travel into the work areas of 
coworkers and see and interact with live images of the coworkers in the environment. In 
20 order to display an image of the user or a coworker in the environment, it is necessary to 
remove any background imagery inconsistent with the computer-generated environment 
from the transplanted image prior to display. For example, if the coworker is in a remote 
office using a computer equipped with software effective to create a virtual environment 
as described by Benman, and the user has a wall, window, bookshelf or other scene in the 
25 background, that information would have to be removed in order to place the person's 
image into the virtual environment in such a way as to create an image of the person 
sitting in the computer generated office environment. 
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However, inasmuch as it would be impractical to require each coworker on the 
network to have a monochromatic (e.g., blue screen) background, there is a need for an 
image processing system or technique which could transplant a desired image from one 
scene into another scene regardless of the background in the first scene. 



SUMMARY OF INVENTION 



10 The need in the art is addressed by the system and method of the present 

invention. Generally, the inventive system includes an optical arrangement (e.g., video 
camera) for providing image data. A memory is provided for storing a first frame of 
image data consisting of a heterogeneous background scene. Next, the user provides to 
the optical arrangement a foreground image, with the same background. This image is 

15 treated as a second frame of image data. . Image processing circuitry extracts the 
foreground imagery from the second frame and strips the background imagery without 
using monochromatic screens or filters. 

Li the preferred embodiment, the image processing circuitry compares picture 
elements of the second frame to corresponding picture elements in said first frame and 

20 replaces each pixel element with a predetermined value if the result of the comparison is 
true and outputting the picture element if the result of the comparison is false. In an 
alternative embodiment, the first frame is subtracted from the second frame and the 
resulting from is filtered and differentiated to provide a template. The template is then 
multiplied against the second frame to extract the desired foreground imagery. 
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BRIEF DESCRIPTION OF THE DRAWINGS 



5 Fig. 1 is a block diagram of an illustrative implementation of an imaging system 

constructed in accordance with the teachings of the present invention. 

Fig. 2 is a diagram which depicts a preferred embodiment of an imaging system 
constructed in accordance with the teachings of the present invention. 

Fig. 3 is a diagram which shows an illustrative application of the teachings of the 
10 present invention in a virtual environment. 

Fig. 4 provides an illustrative frame of background image data as stored by the 
background memory in accordance with the teachings of the present invention. 
Fig. 5 depicts an illustrative camera output of a frame of live image data. 
Fig. 6a illustrates the output of a subtractor utilized by the image processor of the 
15 illustrative embodiment of the invention depicted in Fig 1 . 

Fig. 6b illustrates a differentiated and filled difference image as output by the fill 
II logic utilized by the image processor of the illustrative embodiment of the invention 

depicted in Fig 1. 

Fig. 6c illustrates a foreground isolated image as output by the image processor of 
20 the illustrative embodiment of the invention depicted in Fig 1. 

Fig. 7 is a composite image showing a computer generated virtual environment 
and a live video image transplanted transplanted from the video image of Fig 5. 
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DESCRIPTION OF THE INVENTION 



5 Illustrative embodiments and exemplary applications will now be described with 

reference to the accompanying drawings to disclose the advantageous teachings of the 
present invention. 

Figs. 1 - 3 are provided to disclose the teachings of the present invention. Figs. 4 
- 6 illustrate the images output at various stages of the embodiment of Fig. 1. Fig. 1 is a 
10 block diagram of an illustrative implementation of an imaging system constructed in 
accordance with the teachings of the present invention. The system 10 includes a digital 
video camera 12 adapted to provide single or multiple frames of video data in digital 
form. Numerous companies such as Connectix make such cameras. In the altemative, a 
camera with an analog output may be used if the output thereof is converted to digital 
Q 15 form with an analog to digital converter or other such suitable device. Those skilled in 
^ the art will appreciate that the invention is not limited to the type of digital imaging 

device used. 

;|i The output of the camera 12 is input to a background image memory 14. For this 

image, the user remains outside the field of view of the camera as this memory is used to 
20 stored a frame of static image data that is to be removed from the live video frames as 
discussed more fully below. 

Fig. 4 provides an illustrative frame of background image data as stored by the 
background memory 14. 

In the illustrative embodiment of Fig. 1 , a frame of live image data containing 
25 foreground imagery is stored in a second memory 16. 

Fig. 5 depicts an illustrative camera output of a frame of live image data. This 
frame contains at least a portion of the background image of Fig. 4 partially obscured by 
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an image of a user in the foreground thereof. In accordance with the present teachings, 
the background imagery is removed by an image processor 20 and the foreground image 
is transplanted into another (e.g., computer generated) environment. 

In Fig. 5, a subtractor 16 digitally subtracts signals stored in the first and second 
5 memories representing the color and intensity of individual picture elements (pixels) and 
provides the output to a filter. The effect of the subtraction is to eliminate the 
background imagery leaving a color distorted but image of any foreground imagery 
therein. (In this context, foreground imagery is any imagery not present at the time the 
system was calibrated by capturing the frame of background imagery.) 
10 Fig. 6a illustrates the output of the subtractor 16. This image is processed by the 

filter 24 to remove the color and brightness distortion therein resulting from the 
subtraction process. This image is differentiated to provide an edge defined image. Fill 
logic 28 fills the image, between the edges thereof, with homogeneous values (e.g., 
logical Ts) and logical zeros, outside the image to provide a template as depicted in Fig. 
15 6b. The template is multiplied by the foreground image to provide an output image as 
depicted in Fig. 6c. This image may then be transplanted into another image such as a 
computer generated or stored three-dimensional image as discussed more fully below 
with respect to the illustrative virtual environment application of Fig. 3. 

Rg. 2 is a diagram which depicts a preferred embodiment of an imaging system 
20 constructed inN^cordance with the teachings of the present invention. In the preferred 
embodiment, the sy^em 30 includes the digital camera and background image memory 
14 of Fig. 1. HowewR in the preferred embodiment, the image processor 20 is 
implemented with a comp^tor40, AND gate 42 and microcontroller 50. The 
background image is stored as p^F^ig. 1. However, in the preferred embodiment, the 
25 comparator scans the dynamic live vid^^s^mage and compares it to the stored background 
image on a pixel by pixel basis. If there isVmatch, the comparator 40 outputs a logical 
one, which is inverted at the input to the ANBl gate 42. If there is no match, the 
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comparator outputs a logical zero, which turns on the AND gate 42. This enables the 
current pixel of live video data to be passed by the AND gate as the output of the image 
processor 2uNsThose skilled in the art will appreciate that system would be clocked for 
proper timing analatehes and other delay elements may be required for this purpose as 
5 well. These elements mtijc^e provided by one of ordinary skill in the art as needed for a 
given application without unduNexperimentation. A controller 50 controls each element 
of the circuit along with the clock timing in response to user input or software control. 

Fig. 3 is a diagram which shows an illustrative application of the teachings of the 
present invention in a virtual environment such as that disclosed and claimed in the 

10 above-referenced Benman patent, the teachings of which have been incorporated herein 
by reference. In Fig. 3, in a transmitting system 202, the output of the image processor 20 
is input to a virtual environment controller 218. The environment controller 218 controls 
a three-dimensional virtual environment such as that created using VRML (Virtual 
Reality Modeling Language) or other three-dimensional environment controller. The 

15 intuitive mode controller 220 links applications to the^3D objects in the environment in 
the manner described in the Benman patent. A communications controller 222 controls a 
transceiver 224 so that it may be used for transmission and reception of information as 
necessary to provide a functional virtual environment. 

In a receiving system 204, the signal is received over a communications link such 

20 as a radio, optical or direct (cable, intranet, extranet or intemet) connection. The 
communications controller 228 passes signal representing the image and the location 
thereof received from the transmitting system 202 to an intuitive mode controller 230. 
The virtual environment controller 232 provides application linking in response to the 
user's movements under the control of the intuitive mode controller 230. A virtual 

25 interface 238 processes inputs from a variety of sources including a user and adjusts a 
three-dimensional display 240 accordingly. 
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Fig. 7 is a composite image showing a computer generated virtual environment 
and a live video image transplanted transplanted from the video image of Fig 5. As 
depicted in Fig. 7, the image extracted from the image processor 20 is presented in the 
display as an object in a proper position in the virtual environment based on either a 
default or startup position of the sender as adjusted by any navigation of same. The 
receiver sees the image of the sender without the sender's real world background. 

Thus, the present invention has been described herein with reference to a 
particular embodiment for a particular application. Those having ordinary skill in the art 
and access to the present teachings will recognize additional modifications, applications 
and embodiments within the scope thereof. For example, those skilled in the art will 
appreciate that the teachings of the present invention are not limited to a virtual 
application. The present teachings may be used in any application where a 
transplantation of a live image from a static background to another is desired. 

It is therefore intended by the appended claims to cover any and all such , 
applications, modifications and embodiments within the scope of the present invention. 

Accordingly, 



WHAT IS CLAIMED IS: 



